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AN ABS ALGORITHM FOR A CLASS OF SYSTEMS OF 
STOCHASTIC LINEAR EQUATIONS 

Hai-Shan HarJ3, Antonino Del Popold^l and Zun-Quan Xia[j 



Abstract. This paper is to explore a model of the ABS Algorithms dealing with 
the solution of a class of systems of linear stochastic equations A£ = r/ when rj 
^sO [ is a m-dimensional normal distribution. It is shown that the stepsize on is dis- 

tributed as N(v,i,<Ji) (being u, the expected value of a, and Gi its variance) and 
the approximation to the solutions £j is distributed as N n (Ui,T,i) (being Ui the 
expected value of £j and E$ its variance) , for this algorithm model. 



Key words: ABS algorithm, stochastic linearly system of equations, distribu- 
tion, probability. 



CZ5 . 

AMS Subject Classification (2000): 60H35 65H10 65F10. 



> 

vo : 1 Introduction 



Since the early fifties, the theory of stochastic equations, and in particular stochastic 



integral equations, has been the stage for intense activity. This has been stimulated 
in part by an immense wealth of applications beginning with the Langevin equation 



in statistical mechanics and Wiener's emphasis on the role of randomness in the 
problems of cybernetics. Applications of random equations are now found in many 
areas of engineering, physical and biological sciences and systems theory. 
Historically the study of random equations (stochastic integral equations, stochastic 
operator equations, stochastic difference equations, etc.) started with that of integral 
equations. The theory of the last has grown along two parallel branches. In one, 
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initiated by K. Ito in 1951, the source of randomness is a white noise term leading 
to an important class of stochastic integral equations which has developed hand in 
hand with the theory of Markov processes. In the other, classical linear and nonlinear 
integral equations with random right-hand sides, random kernels or defined on random 
domains are studied. 

The application of stochastic equations is more extensive than deterministic equa- 
tions, because random factors are usually included in many natural phenomena. 
The study of random linear equations can at least be traced back to the works of 
[Bellman 54], [Rosenbloom 55], [Kalman 62]. Random linear difference equations 
arises naturally in many areas of engineering sciences including automatic control 
and signal processing, and have drawn much attention from researchers in the past 
several decades [Brandt 86]. 

Attempts were made by many scientists and mathematicians to develop and unify the 
theory of random equations employing concepts and methods of probability theory 
and functional analysis. The Prague School of probabilists under Spacek initiated a 
systematic study using probabilistic operator equations as models for various systems. 
This development was further energized by the survey article by [BhRe 72] on various 
treatments of random equations under the framework of functional analysis. 
Several techniques have been proposed to solve stochastic linear equations (see for 
instance [Brandt 1986], [Verma 1996]). Stability problems of random linear equations 
have also been studied in [Guo 90], [Guo 93], [GuoLj 95]. 

An approach that has never been tried to solve stochastic linear equations is through 
the use of ABS methods, introduced by Abaffy, Broyden and Spedicato in 1984 in 
the form of a large class of methods for solving linear determined or underdetermined 
systems. The original formulation of 1984 has been later generalized giving a larger 
class of methods that essentialy contains all possible methods building a sequence 
of approximations Xi to the solution having the following property: for X\ arbitrary, 
the vector x m+ i solves the given system. ABS methods have been applied the least 
squares problem, to non-linear equations, to linear and non-linear programming and 
eigenvalue problems. 

Before showing how the ABS algorithm can be used to solve a system of stochastic 
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equations of the form 

A^(w) = V (w) (1.1) 

where £(w), r](w) are random vectors, we recall the ABS algorithm. 
Consider the general linear system, 

Ax = b or ajx = bi 1 i = l,---,m (1-2) 

where x G R m , be R n , m<n and A G iR mxn , A = (a 1; a 2 , • ■ ■ , a m ) T . 
The class of ABS algorithms to solve Eq. (jl.2p was originally introduced by Abaffy, 
Broyden and Spedicato (1984), see [AbBs84] and [AbSp89]. The iterate scheme of 
the basic ABS class of algorithm is defined as follows: 

Basic (unsealed) ABS Class of Algorithms: [AbBs84] , [AbBs89]: 

(A) Initialization. 

Give an arbitrary vector xi G M n , and an arbitrarily nonsingular matrix H\ G 
R n ' n . Set i = 1. 

(B) Compute = Hidi and T{ = r T ei = afxi — b T ei 

(C) Check the compitability of the system of linear equations. 
If Si ^ then goto (D). 

If Si = and Tj = then set x i+ i = x i: H i+1 = Hi and goto (F), the z-th 
equation is a linear combination of the previous equations. Otherwise stop, the 
system has no solution. 

(D) Compute the search vector pi G M" by pi = Hjzi where Z{, the parameter of 
Broyden, is arbitrary satisfying the condition zfHiCii ^ 

(E) Update the approximation of the solution Xi by Xi+i = Xi — ctiPi, where the 
stepsize «j is computed by a« = Ti/afpi Hi — m stop; x m+ i solves the system. 

(F) Update the (Abaffian) matrix Hi. Compute H i+ i = Hi — HidiwJ Hi/ wj Hidi 
where Wi G M n , the parameter of Abaffy, is arbitrary satisfying the condition 

wj Hidi = 1 or 7^ 
3 



(G) Increment the index i by one and goto (B). 



We define n by i matrices A i} Wi and by 

Ai = (at, ■ ■ ■ , ai) T , Wi = (wi, ■ ■ ■ , Wi), Pi = (px, ' • • , Pi) 
Some properties of the above recursion (see for instance, Abaffy and Spedicato (1989), 

|AbSp 89] ), that are the basic formulae to be used later on, are listed below . 

a. Implicit factorization property AjPi = Li with L, nonsingular lower triangular. 

b. Null space characterizations 

N{H i+1 ) = 1Z(AJ), U{HJ +1 ) = K(WA, 
N{A) = K{HJ +1 ) 

where Af= Null and 7£=Range. 

c. The linear variety containing all solutions to Ax = b consists of the vectors of 
the form x = x i+ i + Hf +1 q where q G M n is arbitrary 

As we shall show in the rest of the paper, this method can be used to solve a linear 
stochastic system in the following way. For a linear system 

Ai = r, 

with a m x n matrix of arbitrarary rank with m < n, the so-called ABS algorithm 
produces a solution 

£ = A'r], 

with A 1 some generalised inverse. Now it is quite clear that if r] is a normally dis- 
tributed vector, so is £, or in other words in the original system there are also normally 
distributed vector so is £ as solutions. The mean of £ can obviuosly be computed from 
the mean of rj in the usual deterministic way (save for a component in the kernel of 
A). So the only interesting question left is how to compute the covariance of £ from 
that of r\. Formally it is A I V{A I ) T if the covariance of 77 is V . The paper gives a re- 
cursive formula through the ABS algorithm for the special case that the components 
of 77 are uncorrelated, i. e. V = I. 

A 



This paper is organized as follows. In Section 2, some basic definitions and opera- 
tions that will be used below are presented. A basic ABS algorithm, called ABS-S for 
stochastic linear equations is given in Section 3. Some properties of the approxima- 
tions to solutions and ojj, are given in section 4. In section 5, the ABS-S algorithm 
is illustrated by an example. In section 6, «j is discussed and same results are also 
given. 

2 Preliminaries 

Consider a class of systems of stochastic linear equations being of the form 

A£(w) = r ] (w) (2.1) 

In this paper, we present an ABS algorithm under the condition that r](w) has a 
m-dimensional normal distribution. It will be shown that the iteration step length 
ctj ~ N(ui, <Ji) (being Ui the expected value of ctj and <7j its variance) and the iteration 
solution £j is jV n (C/j,Ej), i=l,2,...m+l. 

We show that (««, Oi) and (Ui, Sj) are determined by four iterative formulae, that £ i+ i 
is a solution of the first % equations and we discuss the step length ctj. 

Definition 2.1[GoMi84] Let (£l,A,V) be a probability space and let £: f2 — > R be 
a real-valued function on Q. We say that £ is a random variable on Q iff for every 
z G -R, {cj : £(u;) < x} e A. 

Definition 2.2 If a random variable £ has a normal distribution with expectation 
and variance a 2 , then it is denoted by £ ~ N(/i, a 2 ). 

Definition 2.3[GoMi84] rj = (rji, r]2, Vm) has a joint N(pL, E) density iff there ex- 
ists an m x n matrix A(m < n) of rank m such that 77 = A£ + /i and £ = (£1, £2, £ n ) 
is a vector of independent N(0,1) random variables. Equivalently £ is a N(0, I n ) mul- 
tiple normal density. 



Definition 2.4 A random variable of random distribution £ is said to be equal to the 
random variable of random distribution 77, denoted by £ = 77, if the distributions of £ 
and 77 are the same and E£=Er] and D£=Dt7, where E and D stands for expectation 
value and variance, respectively. 

Definition 2.5 Consider a system of random linear equations A£=?y. Suppose that 
r) has a certain distribution. If the distribution of £, denoted by £ , is given, then £ 
is a solution of equations A^—rj. 

Property 2.1[GoMi84] If £ = 77 + a, a G then E£=E7?+a, D£=Dt?. 

Property 2.2[Ande 84] If (£,77) is a 2-dimensional normal distribution, then a£ + 677 
is also a normal distribution, where a and b are constants. 

3 ABS-S Algorithm to solve System of Stochastic 
Linear Equations 

Consider a system of stochastic linear equations 

M = n (3.1) 
where 77 = (771, 772, • • • , 77 m ) T is a m-dimensional stochastic vector and A — (ai, 02, • • • , a m ) T G 

The ABS-S algorithm, based on the basic ABS algorithm, is defined, as follows. 
ABS-S Algorithm to solve systems of stochastic linear equations 

(Al) Initialization. 

Give an arbitrary stochastic vector ^ G M n , and an arbitrarily nonsingular 
matrix H x G R n ' n . Set i = 1 

(Bl) Compute the two quantities: 



(CI) Check the compatibility of the system of linear equations. 
If Si ^ then goto (D). 
If Si = and Prob.{ji = 0) = 1 then set 

and go to (F), the i-th equation is a linear combination of the previous equations. 
Otherwise stop, the system has no solution. 

(Dl) Compute the search vector pi e M n by pi = Hf Zi where Zi, the parameter of 
Broyden, is arbitrary satisfying the condition 

zjHidi ^ 

(El) Update the random approximation £j of a solution by 

ii+i = & - 

where the stepsize aj is computed by 

a 



at Pi 

Hi — m stop; £ m+ i solves the system (j3.ip . 

(Fl) Update the ( Abaffian ) matrix Hi. Compute 

HjaJwjHi 

where Wi G iR n , the parameter of Abaffy, is arbitrary satisfying the condition 



w i Hidi = 1 or 7^ 
(Gl) Increment the index i by one and go to (Bl). 
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4 The properties of and ^ 

Let 77 ~ N m (v, I m ) where 77 = (771, 772, , ?7m) T and tj = (tji, tj 2 , v m ) rj 

Proposition 4.1 

ri = af£i - 771 ~ N(a[^ 1 -vi,l) 

OL\ = —f — ~ iv(- 



Proof. From 77 ~ N m (v,I m ), we know that 77^ ~ N(v i: l) : i = 1,2, ...,m. By 
property 2.2, ri has a normal distribution and 77 ~ N(v,I m ), i]i ~ iV(t>j,l),i = 

1,2,.. .,771 

Sri = £(af6 - 771) = af£i - tj i; 

Dn = D{a T l i l -r ll ) = l- 
ui = Ea x = E((a\ Pi) _1 Ti) = (af pi) _1 (a^i - ^i); 
a^ZM = ^((afpO-Vi) = (af Pl )- 2 . 

Therefore 

ai = (afjj>i) _1 ri ~ A^(mi,o-i) 
ri = afe -771 ~ NiEruDn) 

The proof is completed. 

Proposition 4.2 If £ 2 is generated by the ABS-S algorithm, then 



AT (C ~ V _± glgl \ 

aiPi {alPi) 2 

Proof. By using Proposition 4.1, we have that oc\ ~ N(Ea±, Da±). By property 



2.2, since £ 2 = £1 — a.\p\ has a normal distribution and 

U 2 = E& = E(i l -a lPl ) = E(i 1 )-E{a 1 p l )=i 1 -{a T lPl y\a T l i l -v l )p 1 
£ 2 = D£ 2 = Dfa - aiPi) = (afpi)~ 2 pipf 



it follows that 

£ 2 ~iV([/ 2 ,£ 2 ) 

The proof is completed. 
Proposition 4.3 diPj = 0, i < j 

Proof, see |AbBs 84] . |AbSp 89 

Proposition 4.4 If is generated by the ABS-S algorithm, then Tj has a normal 
distribution, and 

n ~ iV(af £i - aJ^jlllEa^pj - v h 1 + af^r^coviajPj, a k p k )ai), i > 2 

where 

cov{ajPj,a k p k ) = p jk p T j p j plp k D(a j )D(a k ) 
where pj k is the correlation coefficient of ctjPj and a k p k 



Proof. By the ABS-S algorithm, we have that = af^i ~ Vii &+1 — & ~~ a iP« 
therefore 

Ti = - »7i = - Qtj-iPi-i) - 7ft = ... = af (£i - E}~i(ajPj)) - Tfc 

Because otj, j < i has a normal distribution, property 2.2, 77, ~ Nty, 1) hence Tj has 
a normal distribution, and 



Er t = E(aJ^ 1 -^J 1 (a J p 3 ))-r ]l ) = aJ^ 1 -J: J r\(Ea J )p J )-v l 
Dt, = D(af (£1 - S}~i(aypj)) ~ ^) = a fPjii C0 KaiPi> a *P*)]°i + L 
The proof is completed. 



Proposition 4.5 If aij is generated by the ABS-S algorithm, then 

AT , a i& ~ a J E j^\( Ea j)Pj ~ v i a i[^~k=i C0V ( a jPj> a kPk)}ai + 1 

a< ~ iV( ^ , - ( T V2 , 1 > 2 

at Pi «Pi) 2 

Proof. Since ojj = {afpi) -1 ^ and property 2.2, we have that the proposition holds. 
The proof is completed. 
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Theorem 4.1 7/6+1 is generated by the ABS-S algorithm, then 

6+1 ~ #n(fi - ZjZKEajjpj, Y>\ k=1 cov{atfj, a k p k )) 

or 

6+1 ~ N n (E£i - (Eaijpi, + piD(ai)pJ - 2cou(&, <XiPi)),i > 2 

Proof. Since 6 = 6-i-«i-iPi-i = 6-2-ai-2Pj-2-ai-iPi-i = ■■■ = £,i-Yi'p^ 1 a j p j , 
and otj, j — 1, 2, . . . , i — 1 have a normal distribution, we have that 6 has a normal 
distribution and 

£>6 = £>(6 - spr 1 ^)?*) = ^ k =i cov ( a jPj, a kPk) 

in terms of property 2.2. The proof is completed. 
Theorem 4.2 6+i ^ s the solution of the first i equations 

Proof. Since 6+i — 6 ~~ one has that af 6+i — a f (6 — — Vi- If I < i, 
then one has that af6+i — a f (6 — Q; i£>i) — a f6- Therefore, E(af^ i+1 ) = E(af£i) and 
D(af£ i+1 ) = D(af£i), one has 

£(af6+i) = £(af6) = • • • = E(af^) = v h 
D{aJi l+l ) = D{aJi t ) = ... = D(af^) = ... = D(a^) 
The proof is completed. 

Proposition 4.6 The ABS-S algorithm terminates in finite steps. 
Proof. The assertion comes directly from Definition (2.5), i.e. according to the 
definition of a solution of the system of the stochastic linear equations At; = rj. The 
proof is completed. 
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5 Conclusion and Discussion 



5.1 On the step length 
Let 

afPi 

1 + of T, l r£ =l cov{otjPj, a k p k )ai 
{at Pi) 2 

It follows that 

a, ~ N{Ea u y/Dal) 

and hence 

P(Ecti - y/Dai < a, < Eoti + a/Do~) = 0.6827 (5.1) 

P(£a i -2 v / ^<a t <£a i + 2 v / I>^) = 0.9545 (5.2) 

P(£«i - 3a/Z>^~ <oti< Eon + 3a/Z>o~) = 0.9973 (5.3) 

Therefore, we obtain that 

(Xi G [Eoii — 3 a/ Daj, -Eaj + 3 a/ -Dctj] 

in other words, on are determined by the initial vector £i, and the first component vi 
of r] as well as afpj. 

5.2 Main results 

1. the system of stochastic linear equations (11.11) with rj ~ N m (v, I m ) can be solved 
by the ABS-S algorithm under some assumptions. 

2. The solution £j generated by the ABS-S algorithm has a normal distribution, if 
T) ~ N m (v, I m ). 

3. The step length ctj generated by the ABS-S algorithm has a normal distribution, 
if rj ~ N m (v,I m ). 

4. Since the matrix A is non-random, generated by ABS-S algorithm are 
non-random, thus the ABS-S algorithm to solve At; = rj has some properties 
that are the same as in the case of the basic ABS algorithm. 
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5.3 Open problems 



1. Suppose that r] has a normal distribution, but the components of r] are depen- 
dent. An open problem is whether the system A£ = r\ has a solution, and if it 
has whether it can be solved by the ABS-S algorithm. 

2. Suppose that the components of i] have different distributions. Another open 
problem is whether the system A£ = r\ has a solution, and if it has whether it 
can be solved by the ABS-S algorithm. 

3. Let A be a random matrix, another open problem is whether the system At; = rj 
has a solution, and if it has whether it can be solved by the ABS-S algorithm. 

4. Suppose rj is a function of a random variable. Another open problem is whether 
the system At; = i] has a solution, and if it has whether it can be solved by the 
ABS-S algorithm. 
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